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DOMINANT EDGE IDENTIFICATION FOR EFFICIENT 
PARTITION AND DISTRIBUTION 

DESCRIPTION 

RELATED APPLICATION 

5 The present application is related to U.S. Patent Application No. 09/ 

(Attorney Docket No. YOR9-2000-0293-US1) entitled "INDEPENDENT NET TASK 
IDENTIFICATIGN FOR EFFICffiNT PARTITION AND DISTRIBUTION^' to 

Kimelman et al.; U.S. Patent Application No. 09/ (Attorney Docket No. 

YOR9-2000-0464-US1) entitled "MACHINE CUT TASK IDENTIFICATION FOR 
jJ 1 0 EFFICIENT PARTITION AND DISTRIBUTION" to Rajan et al. ; and U. S. Patent 

Application No. 09/ (Attorney Docket No. YOR9-2000-0465-US 1) entitled 

i "NET ZEROING FOR EFFICIENT PARTITION AND DISTRIBUTION' to Roth et al., 

U all filed coincident herewith and assigned to the assignee of the present invention. 

^ BACKGROUND OF THE INVENTION 

15 Field of the Invention 

The present invention generally relates to distributed processing and more 
particularly, the present invention relates to efficiently assigning tasks across multiple 
computers for distributed processing. 
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Background Description 

Any large, multifaceted project, such as a complex computer program, may be 
segmented into multiple smaller manageable tasks. The tasks then may be distributed 
amongst a group of individuals for independent completion, e.g., an engineering design 
5 project, distributed processing or, the layout of a complex electrical circuit such as a 

microprocessor. Ideally, the tasks are matched with the skills of the assigned individual 
and each task is completed with the same effort level as every other task. However, with 
such an ideal matched task assignment, intertask communication can become a bottleneck 
to project execution and completion. Thus, to minimize this potential bottleneck, it is 
10 important to cluster together individual tasks having the highest level of communication 
with each other. So, for example, in distributing eight equivalent tasks to pairs of 
individuals at four locations, (e.g., eight design engineers in four rooms) optimally, pairs 
of objects or tasks with the highest communication rate with each other are assigned to 
individual pairs at each of the four locations. 

15 Many state of the art computer applications are, by nature, distributed applications. 

End-users sit at desktop workstations or employ palmtop information appliances on the 
run, while the data they need to access resides on distant data servers, perhaps separated 
from these end-users by a number of network tiers. Transaction processing applications 
manipulate data spread across muhiple servers. Scheduling applications are run on a 

20 number of machines that are spread across the companies of a supply chain, etc. 

When a large computer program is partitioned or segmented into modular 
components and the segmented components are distributed over two or more machines, 
for the above mentioned reasons, component placement can have a significant impact on 
program performance. Therefore, eflSciently managing distributed programs is a major 

YOR9-2000-0466-US1 

2 



challenge, especially when components are distributed over a network of remotely 
connected computers. Further, existing distributed processing management software is 
based on the assumption that the program installer can best decide how to partition the 
program and where to assign various program components. However, experience has 
5 shown that programmers often do a poor job of partitioning and component assignment. 

So, a fundamental problem facing distributed application developers is appUcation 
partitioning and component or object placement. Since communication cost may be the 
dominant factor constraining the performance of a distributed program, minimizing 
inter-system communication is one segmentation and placement objective. Especially 

10 when placement involves three or more machines, prior art placement solutions can 

quickly become unusable, i.e., what is known as NP-hard. Consequently, for technologies 
such as large application frameworks and code generators that are prevalent in 
object-oriented programming, programmers currently have little hope of determining 
effective object placement without some form of automated assistance. En masse 

15 inheritance from towering class hierarchies, and generation of expansive object structures 
leaves programmers with little chance of success in deciding on effective partitioning. 
This is particularly true since current placement decisions are based solely on the classes 
that are written to specialize the framework or to augment the generated appUcation. 

Furthermore, factors such as fine object granularity, the dynamic nature of 
20 object-based systems, object caching, object replication, ubiquitous availability of 

surrogate system objects on every machine, the use of factory and command patterns, etc., 
all make partitioning in an object-oriented domain even more difficult. In particular, for 
conventional graph-based approaches to partitioning distributed applications, fine-grained 
object structuring leads to enormous graphs that may render these partitioning approaches 
25 impractical. - 
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Finally, although there has been significant progress in developing middleware and 
in providing mechanisms that permit objects to inter-operate across language and machine 
boundaries, there continues to be little to help programmers decide object-system 
placement. Using state of the art management systems, it is relatively straightforward for 
5 objects on one machine to invoke methods on objects on another machine as part of a 
distributed application. However, these state of the art systems provide no help in 
determining which objects should be placed on which machine in order to achieve 
acceptable performance. Consequently, the initial performance of distributed object 
applications often is terribly disappointing. Improving on this initial placement 
10 performance is a diflBcult and time-consuming task. 

Accordingly, there is a need for a way of automatically determining the optimal 
program segmentation and placement of distributed processing components to minimize 
communication between participating distributed processing machines. 



SUMMARY OF THE INVENTION 

15 It is therefore a purpose of the present invention to improve distributed processing 

performance; 

It is another purpose of the present invention to minimize communication between 
distributed processing machines; 

It is yet another purpose of the invention to improve object placement in 
20 distributed processing applications; 

It is yet another purpose of the invention to determine automatically how objects 
should best be distributed in distributed processing applications 
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it is yet another purpose of the invention to minimize communication between 
objects distributed amongst multiple computers in distributed processing applications. 

The present invention is a task management system, method and computer 
program product for determining optimal placement of task components on multiple 
5 machines for task execution, particularly for placing program components on multiple 
computers for distributed processing. First, a conmiunication graph is generated 
representative of the computer program with each program unit (e;g., an object) . 
represented as a node in the graph. Nodes are connected to other nodes by edges 
representative of communication between connected nodes. A weight is applied to each 

10 edge, the weight being a measure of the level of communication between the connected 
edges. Terminal nodes representative of the multiple computers are attached to the 
communication graph. Then, dominant edges are identified within the communication 
graph. For any non-terminal node, a connected edge is dominant if it is at least as heavy 
(its weight is greater than or equal to) as the sum of the remaining non-terminal edges and 

1 5 the heaviest of the remaining terminal edges. The min cut for the communication graph 
need not include any dominant edges and so, dominant edges are removed fi"om 
consideration for the final min cut solution. Finally, program components which may be a 
single program unit or an aggregate of units are placed on computers according to the 
communication graph min cut solution. 



20 BRIEF DESCRIPTION OF THE DRAWINGS 
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The foregoing and other objects, aspects and advantages will be better understood 
from the following detailed preferred embodiment description with reference to the 
drawings, in which: 

Figure 1 shows an example of a flow diagram of the preferred embodiment of the 
5 present invention wherein a program is segmented, initially, and initial segments are 

distributed to and executed on multiple computers; 

Figures 2A-C show an example of a communication graph; 

Figure 3 is a flow diagram of the optimization steps for determining an optimum 
distribution of program components; 
10 Figure 4 shows an example of a portion of a communication graph reducible by the 

preferred embodiment Dominant Edge method; 

Figure 5 is an example of the steps of identifying dominant edges; 

Figure 6 is an example of the steps in contracting a dominant edge. 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS OF THE 
15 INVENTION 

As referred to herein, a communication graph is a graphical representation of a 
multifaceted task such as a computer program. Each facet of the task is an independent 
task or object that is represented as a node and communication between tasks or nodes is 
represented by a line (referred to as an edge) between respective communicating nodes. 

20 Participating individuals (individuals receiving and executing distributed tasks) are referred 
to as terminal nodes or machine nodes. A net is a collection of nodes connected together 
by edges. Two nets are independent if none of the non-terminal nodes of one net shares 
an edge with a non-terminal node of the other. Thus, for example, a communication graph 
of a computer program might include a node for each program object and edges would be 

25 between communicating objects, with edges not being included between objects not 
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communicating with each other. In addition, a weight indicative of the level of 
communication between the nodes may be assigned to each edge. Graphical 
representation and modeling of computer programs is well known in the art. 

Referring now to the drawings, and more particularly, Figure 1 is an example of a 
5 flow diagram 100 of the preferred embodiment of the present invention wherein a program 
is segmented, initially, and initial segments are distributed to and executed on muhiple 
computers. First, in step 102 the communication patterns of a program are analyzed to 
form a communication graph. Then, in step 104, the traces of the communication graph 
are analyzed, and an initial partition is determined. In step 106, the partition is optimized 
10 for minimum interpartition conmiunication. In step 108, the individual objects are 

distributed amongst participants for execution according to the optimize partition of step 
106. 

A component refers to an independent unit of a running program that may be 
assigned to any participating individual, e.g., computers executing objects of a distributed 

15 program. Thus, a component may refer to an instance of an object in an object oriented 
program, a unit of a program such as Java Bean or Enterprise Java Bean or, a larger 
collection of program units or components that may be clustered together intentionally 
and placed on a single participating machine. Further, a program is segmented or 
partitioned into segments that each may be a single component or a collection of 

20 components. After segmenting, analyzing the segments and assigning each of segments or 
components to one of the multiple participating machines or computers according to the 
present invention, the final machine assignment is the optimal assignment. 

Thus, a typical communication graph includes multiple nodes representative of 
components with weighted edges between communicating nodes. Determining 
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communication between components during program execution may be done using a 
typical well known tool available for such determination. Appropriate communication 
determination tools include, for example, Jinsight, for Java applications that run on a 
single JVM, the Object Level Tracing (OLT) tool, for WebSphere applications or, the 
5 monitoring tool in Visual Age Generator. 

Figures 2A-C show an example of a communication graph of a net 110 that 
includes multiple nodes 112, 114, 116, 118, 120 and 122. Each node 112, 114, 116, 118, 
120 and 122 represents a program component connected to communication edges 124, 
126, 128, 130, 132, 134, 136 and 138 to form the net 110. Adjacent nodes are nodes that 
10 share a common edge, e.g., nodes 1 14 and 122 share edge 126. Each edge 124, 126, 128, 
130, 132, 134, 136 and 138 has been assigned a weight proportional to, for example, the 
number of messages between the adjacent components. 

In Figure 2B, Machine nodes 140, 142 and 144 representative of each participating 
machine (three in this example) are shown connected by edges 146, 148, 150. Initially, a 

15 node 1 12, 1 14, 1 16, 1 18, 120 and 122 may be placed on a machine 140, 142, 144 by 

adding an edge 146, 148, 150 with infinite weight (indicating constant communication) 
between the node and the machine. Typically, initial assignment places nodes with specific 
fiinctions (e.g., database management) on a machine suited for that fianction. After the 
initial placement assigning some nodes 112, 114 and 122 to machines 140, 142, 144, other 

20 nodes 1 16, 1 18, 120 are assigned to machines 140, 142, 144, if they communicate heavily 
with a node 112, 114, 122 already assigned to that machine 140, 142, 144. Additional 
assignment is effected by selectively collapsing edges, combining the nodes on either end 
of the collapsed edge and re-assigning edges that were attached to one of the two former 
adjacent nodes to the combined node. When assignment is complete, all of the nodes 1 12, 

25 1 14, 1 16, 1 18, 120 and 122 will have been placed on one of the machines at terminal 
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nodes 140, 142, 144 and the final communication graph may be represented as terminal 
nodes 140, 142, 144 connected together by communication edges. 

For this subsequent assignment, the graph is segmented by cutting edges and 
assigning nodes to machines as represented by 152, 154 and 156 in Figure 2C to achieve 
5 what is known in the art as a minimal cut set or min cut set. A cut set is a set of edges 

that, if removed, eliminate every path between a pair of terminal nodes (machine nodes) in 
the graph. A min cut set is a cut set wherein the sum of the weights of the cut set edges is 
minimum. While there may be more than one min cut set, the sum is identical for all min 
cut sets. A min cut may be represented as a line intersecting the edges of a min cut set. 

10 So, in the example of Fig. 2C, the sum of the weights of edges 124, 126, 128, 132 and 138 

is 2090, which is cost of the cut and is representative of the total number of messages that 
would be sent between machines at terminal nodes 140, 142, 144 with this particular 
placement. The min cut identifies the optimum component placement with respect to 
component communication. While selecting a min cut set may be relatively easy for this 

1 5 simple example, it is known to increase in diflBculty exponentially with the number of 

nodes in the graph. 

Figure 3 is a flow diagram 160 of the optimization steps for determining an 
optimum distribution of program components to individual participating computers 
according to a preferred embodiment of the present invention. First, in step 162, an initial 

20 communication graph is generated for the program. Then, in step 164 machine nodes are 

added to the communication graph. As noted above, certain types of components are 
designated, naturally, for specific host machine types, e.g., graphics components are 
designated for clients with graphics capability or, server components designated for a data 
base server. After assigning these host specific components, in step 168 independent nets 

25 are identified and the communication graph is partitioned into the identified independent 
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nets as described in U.S. Patent Application No. 09/ (Attorney Docket No. 

YOR9-2000-0293-US1) entitled "INDEPENDENT NET TASK IDENTIFICATION 
FOR EFFICIENT PARTITION AND DISTRIBUTION" to Kimelman et al., assigned to 
the assignee of the present invention and incorporated herein by reference. In step 170 a 
5 min cut is identified for each of the independent nets, using the preferred embodiment 

Dominant Edge reduction method to reduce the independent nets and identify the min cut 
according to the present invention described hereinbelow. The min cuts for all of the 
independent nets are the min cut for the whole communication graph. 

Figure 4 shows an example of a portion of a communication graph 180 reducible 
10 by the preferred embodiment Dominant Edge method of the present invention. In this 
example, the graph portion 180 includes four (4) non-terminal nodes 182, 184, 186 and 
188 and four terminal nodes 190, 192, 194 and 196. Node 182 is connected, respectively, 
to non-terminal nodes 184, 186 and 188 by edges 198, 200 and 202, referred to as 
non-terminal edges, and to terminal nodes 190, 192, 194 and 196 by edges 204, 206, 208 
1 5 and 210, referred to as terminal edges. 

According to the present invention, for any non-terminal node a connected iedge is 
dominant if it is at least as heavy (its weight is greater than or equal to) as the sum of the 
remaining non-terminal edges and the heaviest of the reriiaining terminal edges. Further, 
every communication graph contains a min cut solution that does not include dominant 

20 edges. Thus, dominant edges may be contracted. So, for node 182 in this example, edge 
200 is the heaviest edge (30) and edge 204 is the heaviest terminal edge (20). The sum of 
the remaining non-terminal edges 198 and 202 and edge 204 is 25 (2 + 3 +20) which is 
less than edge 200. Thus, non terminal edge 200 is a dominant edge and may be 
collapsed combining nodes 182 and 186 to reduce the graph and, therefore, simplify the 

25 min cut solution somewhat. 
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Figure 5 is an example of the steps 220 of identifying dominant edges according to 
the preferred embodiment of the present invention. First, in step 222, non-terminal nodes 
adjacent to at least two or more terminal nodes are identified. In step 224 the heaviest 
edge is identified for each node. Then, in step 226, nodes are identified that potentially are 
5 connected to a dominant edge as those nodes that, when excluding the heaviest edge, are 
still adjacent to at least two or more terminal nodes. In step 228, again excluding the 
heaviest edge, the weight of the heaviest terminal edge is added to the sum of the weights 
of the non-terminal edges. In step 230, that sum is compared against the weight of the 
heaviest edge. If the heaviest edge is heavier than the sum, then, the heaviest edge is a 
10 dominant edge and, as such, is not required for inclusion in a min cut solution. Otherwise, 
the heaviest edge is not a dominant edge 232. Each dominant edge is collapsed in step 
234. 

In other words, for any non-terminal node that is adjacent to three (3) or more 
terminals nodes, if the heaviest of the edges incident to the given non-terminal node is a 

15 terminal edge connecting to a terminal node; then, if the weight of that heaviest edge is 
greater than or equal to the sum of the weights of all non-terminal edges at the 
non-terminal node plus the weight of the second-heaviest terminal edge, then that heaviest 
edge satisfies the condition and is a dominant edge. Likewise, if the non-terminal node is 
adjacent to two or more terminals nodes, and if the heaviest of the edges incident to the 

20 given non-terminal node is a non-terminal edge; then, if that heaviest edge has \yeight 

greater than or equal to the sum of the weights of all of the remaining non-terminal edges 
plus the weight of the heaviest terminal edge, then that heaviest edge satisfies the 
condition and so, is a dominant edge. A dominant edge is not part of the (only) multiway 
minimum cut of the graph, and as such, may be contracted or collapsed. 
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Figure 6 is an example of the steps in contracting a dominant edge 232, First in 
step 2320 the two nodes connected by the dominant edge are merged, resulting in a single 
merged node that includes the components of both original nodes. Then, in step 2322 the 
dominant edge is discarded. Finally, in step 2324 any "parallel" edge groups (edges 
5 connecting the merged node to the same adjacent node) resulting from the merger are 

replaced with a single edge with its weight equal to the sum of parallel edge weights. 
Thus, as a result of contracting dominant edges, the graph has been reduced wherein a min 
cut solution may be found with less effort. 

In the preferred embodiment, the min cut step 170 is an iterative process, wherein 

10 independent nets are reduced using, when necessary, the Dominant Edge identification 

steps described herein in combination with the Machine Cut method of U.S. Patent 

Application No, 09/ (Attorney Docket No. YOR9-2000.0464-US1) entitled 

"MACHINE CUT TASK IDENTff IC ATION FOR EFFICIENT PARTITION AND 
DISTRIBUTION" to Rajan et al., and the Net Zeroing method of U.S. Patent Application 

1 5 No. 09/ (Attorney Docket No. YOR9-2000-0465-US 1) entitled "NET 

ZEROING FOR EFFICIENT PARTITION AND DISTRIBUTION^' to Roth et al., filed 
coincident herewith, assigned to the assignee of the present invention and incorporated 
herein by reference. Further, as independent nets are reduced, those reduced nets are 
fiarther checked as in step 168 above to determine if they may be divided into simpler 

20 independent nets. Then, the Dominant Edge method of the preferred embodiment is 
applied to those simpler independent nets. To reach a solution more quickly, on each 
subsequent pass, only nodes and edges of a subgraph that were adjacent to areas reduced 
previously are rechecked. Thus, the communication graph is simplified by eliminating 
dominant edges to reach a min cut solution much quicker and much more eflBciently than 

25 with prior art methods. 
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The reduction method of the preferred embodiment reduces the number of 
independent components in the communication graph of a complex program. In the best 
case, an appropriate allocation of every component in the program is provided. However, 
even when best case is not achieved, the preferred embodiment method may be combined 
5 with other algorithms and heuristics such as the branch and bound algorithm or the 

Kemighan-Lin heuristic to significantly enhance program performance. Experimentally, 
the present invention has been applied to communication graphs of components in several 
programs with results that show significant program allocation improvement, both in the 
quality of the final solution obtained and in the speed in reaching the result. 



10 Although the preferred embodiments are described hereinabove with respect to 

distributed processing, it is intended that the present invention may be applied to any 
multi-task project without departing fi"om the spirit or scope of the invention. Thus, for 
example, the task partitioning and distribution method of the present invention may be 
applied to VLSI design layout and floor planning, network reliability determination, web 

15 pages information relationship identification, and "divide and conquer" combinatorial 
problem solution approaches, e.g., "the Traveling Salesman Problem." 

While the invention has been described in terms of preferred embodiments, those 
skilled in the art will recognize that the invention can be practiced with modification within 
the spirit and scope of the appended claims. 
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